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Introduction 


System  of  systems  (SoS),  either  directed  as  a  program,  acknowledged 
as  a  set  of  programs,  or  emergent  as  in  collaborative  or  virtual 
varieties*,  ALL  need  a  way  to  assess  software  performance  (SWP): 

•  Assess  causes  of  SWP  issues 

•  Determine  indicators  and  measures  of  SWP 

•  Plan  SWP  measurement  in  tests 


Fundamental  question:  Will  software  enable  planned  capabilities 
within  end-to-end  field  environment? 


We  provide  a  10-step  method  for  planning/assessing  SWP,  allowing  for 
respective  improvement  of  architecture  and  test  processes 

Our  method  is  based  on  experience  within  a  major  directed  SoS  Service 
Orientated  Architecture  (SOA)  DoD  acquisition  program 


*  See  “Exploring  Enterprise,  System  of  Systems,  and  System  and  Software  Architectures”  by  Paul  C.  Clements,  SEI,  2009. 
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Software  Performance  10-Step  Method 


1  -  Make  SOS  SOA 

r  ^ 

2  -  Review  key 

r  ^ 

3  -  Make  sample 
scenarios: 

r  ^ 

4  -  Make  list  of 
metrics  (indicate 

layout  — > 

performance  view 

resource  limiters 
from  layout 

L. _ J 

What  are  sources 
of  performance 
^  impacts  in  each? 

sources, 

architecture  ties  if 
known) 

5  -  Add  in  required 
SWP  metrics  from 
documents  (quality/ 
best  practice/ 
critical  resources) 


8  -  Use  populated 
metrics  matrix  to 
plan  future  tests 
and  mine  data  from 
existing  data  sets 


6  -  Find  test  events 
that  have  occurred: 
Rate  the  maturity  of 
each  for  each 
metric 


7  -  Circulate 
results/vetting: 
What  metrics  and 
events  are 
missing? 


9  -  Use  architecture 
tie-ins  to  improve 
software 
performance 


10  -  Determine 
repeat  schedule 
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An  Example  SoS  Layout 


System  of  Systems 


This  schematic  represents  the  SoS 
context  in  which  the  example  software 

was  delivered 


Applications  using  services  reside  at 
the  system  level  and  assume  services 
are  instantiated  on  blades 


Processing  Unit 


i 


This  is  one  of  multiple  context  views 
required;  it  was  chosen  to  allow 
further  break  down  of  performance 
affecting  sources 


Software  Engineering  Institute 


Blade 


(Blade)  OS 
Middleware 


Service 


Instance  of 
Discoverable 
Service 
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Notional  SoS  Layout:  On  a  Processing  Unit 


Blade  Server 


Processing  Unit 
-  (or  Rack) 

mrnm  


Blade  in  same 
Processing  Unit 


Fiber  Channel  or  similar 
Interface  to  Shared  RAID 


* 


v 


Faster 


Firewall  +  Router  with  LAN  (Gigabit 
Ethernet  et  al.)  Interfaces 

Slower 


/  This  schematic  provides 
processor  level  SoS 
context  fidelity 
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Notional  SoS  Layout:  A  System 


Processing  Unit 


Note:  The  delay  to  the 
WAN  interface 
processing  units  are  the 
same  but  performance 
will  need  to  add  WAN 
delays  for  each  link 


Faster 


Shared 

RAID 


Processing 
Unit 


System  Firewalk  Router+ 
Radio  Short  Range 


System  Firewalk  Router+ 
Radio  Long  Range 


System  Firewalk  Router+ 
Radio  Satellite 


Slower 


System 

^1 

i 

i 

+Short  Range  Wireless 
/  WAN  Delays 


/ 


+Long  Range  Wireless 
WAN  Delays 

+Satellite  Link 
WAN  Delays 
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Software  Performance  10-Step  Method 


Software/Hardware  Performance  Planning 


Counter  clock-wise, 
faster  to  slower 


DRAM 


FLASH 


-  100  microseconds 


~1  micro¬ 
second 


Human 

Interactions 


1  second 
to  minutes 


~  100 

microseconds 


Blade 


System 
Over  GiG 
(Indirect) 


System 
Over  GiG 
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~1  millisecond 


~  5 

milliseconds 


Notional  Representation 
Blue  =No  data 
Orange  =  Simulated  Data 
Green:  Live  Data 


System  on 
Same 
Platform 


0.1  to  1  second 


System  on 
Different 
Platform 


Designers  should  manage  access 
to  slower  methods  when  possible 
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Scale  Issues 


The  work  of  each  blade  (CPU/memory/ 
LAN  utilization,  middleware,  etc.)  will 
increase  based  upon 

•  total  number  of  systems  in  the 
system  of  systems 

•  how  often  the  users  need 
services  in  other  systems/ 
processing  units/blades 

Each  increase  in  scale  increases 
resource  needs  per  service 
hosting  blade 


Software  Engineering  Institute 
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Software  Performance  10-Step  Method 


r ^ 

3  -  Make  sample 

r  ^ 

4  -  Make  list  of 

1  -  Make  SOS  SOA 

2  -  Review  key 

scenarios: 

metrics  (indicate 

layout  performance 

resource  limiters 

What  are  sources 

sources, 

“1 

view 

from  layout 

of  performance 

architecture  ties  if 

impacts  in  each? 

known)  , 

A  Possible  Scenario  - 1 


User  1  on 
Blade  A  on  PU1 


User  2  on  Blade  B 
on  PU2 


Start 


End 


Instance  of 
Discovered 
Service 


Middleware 

OS 


User  1  Requests  Data  from  User  2 

Where  is  software  performance 
affected  (delayed)? 


System  2 


Instance  of 
Discovered 
Service 
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A  Possible  Scenario  -  2 


What  metrics  affect 
software 
performance  in 
previous  scenario? 

l.  ^  User  1  to  User  2,  examples: 

•  On  Blade  A:  Service  Call  to  Middleware 

•  Delays  Between  Blade  &  Processing  Unit 

•  Delays  on  Short  Range  Router/FW  /Radio  1 

•  Delays  on  Short  Range  Router/FW  /Radio  2 

•  LAN  Latency  From  Short  Range 
Router/FW/Radio  2  to  PU2’s  LAN  Blade 

User  2  to  User  1 :  Reverse  previous  bullet! 


-  Software  Engineering  Institute  Carnede Mellon  J  Wessel  B  Mever  Mav 2010 

- — —  ^  ©2010  Carnegie  Mellon  University 


SSTC  2010 


13 


Software  Performance  10-Step  Method 


Make  a  Software  Performance  Metrics  Matrix 


Consider  the  design  levels  and  requirements 
•  Aid:  ‘desk’  running  scenarios 

from:  intended  use,  take  to  break  (‘rainy  day’),  and  requirements 


The  Initial  Matrix 


•  Metric  name:  title,  short  name  and  key  words  for  tagging 

•  Why  it  should  be  collected,  including  Need  Type 

•  An  example  of  the  ways  to  collect  it:  How? 

•  Any  ties  to  requirements,  directly  or  as  contributors 

•  High-Level  Type:  What  aspect  of  the  overall  design  am  I  assessing? 


# 

Short 

Name 

Metric  Title 

Why? 

Keywords  (for 
Tagging) 

How? 

Need 

Type 

High 

Level 

Type 

1 

Bcalls_ 

Count 

Blade  to  blade  calls 
(tagged  by  service,  by 
process,  by  user,  by 
case/scenario/time 

Limiting  calls  from  blade 
to  blade  reduces  time 
(due  to  bus  use) 

Blade,  calls,  count, 
service,  process 

Bus  monitoring  via 
Processing  Unit 
against  process 
monitor 

Efficiency 

Engineer 

2 

HDCalls_ 

Count 

Service  traffic  count  to 
drives 

Which  services, 
applications,  clients  of 

User,  service,  raid, 
calls 

Process-message 
snapshots  and 

Efficiency 

Engineer 

applications  are  hitting 
the  drives  often.  The 
more  often  RAM  is  used 
in  lieu  of  the  drives,  the 
quicker  the  app  will  run. 


parse  (or  logging 
parse)  for  OS+bus 
capture  (log  parse) 
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Software  Performance  10-Step  Method 


r  ^ 

1  -  Make  SOS  SOA 

2  -  Review  key 

layout  performance 

resource  limiters 

view 

from  layout 

Adding  Metrics  Using  Existing  Matrix  Guidance 


Use  list  of  20  minimums  to  fill  in  list  made  from  scenarios 

•  This  provides  a  set  of  metrics  that  might  not  have  emerged  from  Step  4 
scenarios,  but  come  from  experience  with  similar  systems 

Add  quality  metrics  related  to  software  performance 

Add  guidance  from  requirements  documents 


Sample  Key  Metrics  for  Software  Performance 

# 

Short  Name 

Metric  Title 

Why? 

How? 

1 

HDPartJJt 

Partition/disk  usage  over 
time/scenario/  factor 

Avoid  overfilling  partitions  (which  can 
slow  or  stop  a  system);  determine 
which  situations  stress  disks 

Repeated  capture 
from  OS 

2 

LANUtil 

Platform  LAN  utilization 

Prevent  overuse  of  LAN  on  platform; 
watch  for  processes  that  could  be 
done  in  blade  instead  of  over  LAN 

SNMP  MIB  from 
routers 

3 

RAMJJtil 

RAM  utilization  (by  client, 
service,  application)  overtime 

Prevent  over-utilization,  prevent 
resource  hogging/application 

Repeated  capture 
from  OS 
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Software  Performance  10-Step  Method 


Software 


Testing/Simulation  Types 


Cube  of  ‘Realism’  (Omitting  Network*) 


*  One  could  extend  to  ‘Network’  for  a  4th  Dimension 


Assess  realism  per  test  event 

1 .  Software 

•  Mod=Modeled 

•  Sim=Simulated 

•  Proto=Prototype 

•  EB=Early  Build 

•  LB=Later  Build 

•  Mat=Mature 

2.  Hardware 

•  Sim=Simulated 

•  EP=Early  Prototype 

•  LP=Late  Prototype 

•  IP=lnitial  Production 

•  FP=Full  Production 

3.  Scale 

•  SB/MB=Single  Blade/Multiple  Blades 

•  PU/MPU=Process  Unit/Multiple  PUs 

•  SS=Single  System 

•  LS=Limited  Multiple  System 

•  PS=Partial  Scale 

•  FS=Full  Scale 


|§r  Software  Engineering  Institute 
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Test  for  Realism 


Realism  varies  by  metric  inside  each  test  event  due  to  available 
test  assets  and  timeframes 

Test  targeted  at  reducing  one  set  of  risks  might  collect  data  on  other 
related  areas  as  a  side  effect 

Review  of  full  test  artifacts  can  mine  for  ‘off-target’  collections 

Off-target  metric  collections  might  be  at  a  lower  fidelity  level  than 
metric  included  in  risk  target  of  test 


Software  Engineering  Institute 


Carnegie  Mellon 
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Trending  and  Correlation 


Other  correlations 

•  Regression  comparisons? 

•  Gap  analysis;  compare 
w/desired  performance 

Tie  to  architecture  (design, 
various  levels) 


System  Architecture; 
Software  Architecture 


Which  cross  correlations  have  a  payoff? 
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Software  Performance  10-Step  Method 


r  ^ 

1  -  Make  SOS  SOA 

2  -  Review  key 

layout  performance 

resource  limiters 

view 

from  layout 

/  V 


5  -  Add  in  required 
SWP  metrics  from 
documents  (quality/ 
best  practice/critical 
resources) 


6  -  Find  test  events 

|  that  have  occurred: 

j  Rate  the  maturity  of 
each  for  each 

l  metric 

7  -  Circulate 
results/vetting: 

What  metrics  and  — 
events  are 
missing? 


Who  Vets  the  SWP  Metrics  Matrix? 


Testing  groups  are  usually  scattered  in  various  system  groups  and  at 
program  level 

Bring  representatives  of  each  group  together  to  examine  each  iteration 
of  metrics  matrix 

•  Limit  attendance  to  those  who  understand  test  metrics  and  fidelity  levels 

•  Honesty,  not  spin,  is  important 

•  Get  leadership  backing 

Vet  matrix  with  this  newly-formed  Technology  Interchange  Group  (TIG). 
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Vetted  Matrix  and  Procedure  Linkage 


Use  matrix  as  a  starting  point  for  discussion  for  initial  TIG  meeting 

•  Discuss  matrix  data:  Was  anything  missed? 

-  All  that  has  happened  to  date:  Does  it  include  all  test  events? 

-  Knowledge  of  events  at  each  scale:  Does  it  capture  the  correct  realism  and  scale  of 
each  event? 

•  Revise  matrix 

-  Include  missed  or  incomplete  items  discovered 

-  Gain  consensus  on  correctness/completeness  of  metrics:  Are  we  measuring  the 
right  performance?  Does  the  list  account  for  SWP  issues  that  may  emerge  later? 

Re-circulate  to  confirm  results 

•  Store  matrix  in  configuration-controlled,  commonly  accessible  location 
(Sharepoint,  Wiki,  etc.) 

•  Encourage  TIG  to  comment  and  distribute  to  their  teams  for  comment 

•  Collect  comments,  confirm  veracity  of  updates  with  TIG,  revise  matrix 

Repeat  until  there  is  a  strong  confidence/consensus  in  matrix 
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Software  Performance  10-Step  Method 


1  -  Make  SOS  SOA 

r  ^ 

2  -  Review  key 

r  ^ 

3  -  Make  sample 
scenarios: 

4  -  Make  list  of 
metrics  (indicate 

layout  performance 
view 

. _ J 

resource  limiters 
from  layout 

L _ J 

What  are  sources 
of  performance 
^  impacts  in  each? 

sources, 

architecture  ties  if 
known) 

5  -  Add  in  required  ^ 

6  -  Find  test  events 

7  -  Circulate 

SWP  metrics  from 

that  have  occurred: 
Rate  the  maturity  of 
each  for  each 
metric  , 

results/vetting: 
What  metrics  and 
events  are 
missing?  . 

oocuments  ^quality/ 
best  practice/critical 
resources)  , 

■ 

8  -  Use  populated 
metrics  matrix  to 
plan  future  tests  & 
mine  data  from 
existing  data  sets 


9  -  Use  architecture 
tie-ins  to  improve 
software 
performance 


10  -  Determine 
repeat  schedule 
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Metrics/Planning  for  Metrics  Collection  using 
the  Matrix 


Insert  metrics  with  low  event  coverage  into  future  test  events. 

♦  What  metrics  (rows)  in  the  matrix  have  no  associated  events  (i.e.  empty 
columns)?  Which  metrics  were  only  measured  at  a  low  scale  or  fidelity? 

•  Insert  metrics  into  event  plans  and  insert  planned  events  into  the  matrix 

Make  metric  list  a  standard  minimum  for  tests  at  any  scale 

Create  correlation  standards  and  a  history  of  what  correlations  have 
lead  to  problem  discovery 

Agree  on  initial  conditions  for  tests 
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Ideas  for  Entry  Criteria:  Metrics  Infrastructure 


Consolidated  Metrics  Library  Database 

•  Complex  trends  and  simple  points 

•  Easily  accessible  by  architects/engineers/development/other  test  groups 

•  Metadata  tagging  using  a  standard 

Insert  into  test  schedule 

•  Run  future  test  event  planning  through  TIG 

•  Invite  group  edits  to  matrix 
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Software  Performance  10-Step  Method 


r  ^ 

1  -  Make  SOS  SOA 

2  -  Review  key 

layout  performance 

resource  limiters 

view 

from  layout 

Software  Performance  Management:  A  Team 


Performance  TIG 
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Relating 


to  Metrics 


It  is  useful  with  a  vetted  metrics  matrix  to  tie  each  metric  to  architecture 

•  Use  ties  to  improve  performance 

There  are  likely  no  orphan  metrics;  they  are  just  more  complex  to  trace 
to  architecture  and  design 


Repeated  columns  of  higher  fidelity  and  realistic  events  improve 
confidence  that  the  metric  is  covered  and  performance  quantified; 
use  these  to  plan  tests 


Architecture  and  design  elements  tied  to  performance  will  gain 
confidence  with  successive  events;  again  test  planning 
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Conclusions 


Understanding  software  performance  for  a  SoS  SOA  system  is  complex; 
managers  need  to: 

•  Understand  the  system’s  respective  performance  affecting  levels 

•  Develop  a  metrics  list  derived  from  scenarios  and  other  sources 

•  Tie  in  test  events  to  make  the  metrics  matrix 

•  Have  a  way  to  circulate  the  matrix  by  understanding  the  organization 

•  Feedback  the  matrix  and  metrics  testing  results  to  architecture  leads 

•  Keep  the  matrix  current  or  status  will  be  unknown 
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BACK-UP  SLIDES 
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Acronym  List 


CM 

Configuration  Management 

COTS 

Common  Off  The  Shelf 

CPU 

Central  Processing  Unit 

DoD 

U.S.  Department  of  Defense 

DRAM 

Dynamic  Random  Access  Memory 

E2E 

End-to-End 

FW 

Fire  Wall 

GiG 

Global  Information  Grid 

GUI 

Graphical  User  Interface 

HD 

Hard  Drives 

H/W 

Hardware 

LAN 

Local  Area  Network 

LUT 

Limited  User  Test 

IPT 

Integrated  Process  Team 

M&S 

Modeling  and  Simulation 
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Acronym  List 


OS 

Operating  System 

PU 

Processing  Unit 

RAID 

Redundant  Array  of  Independent  Disks 

RAM 

Random  Access  Memory 

RFP 

Request  For  Proposal 

SE 

Systems  Engineering 

SEC 

Army  Software  Engineering  Center 

SOA 

Service  Oriented  Architecture 

SoS 

System  of  Systems 

SW 

Software 

SWP 

Software  Performance 

TIG 

Technology  Interchange  Group 

TRL 

Technical  Readiness  Level 

WAN 

Wide  Area  Network 
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Services 


“Services  and  applications  are  defined  as  primarily  software  based 
components  which  perform  specific  functions  using  standard 
interfaces.  A  service  is  defined  as  a  mechanism  to  enable  access  to 
one  or  more  capabilities,  where  the  access  is  provided  using  a 
prescribed  interface  and  is  exercised  consistent  with  constraints 
and  policies  as  specified  by  the  service  description  (reference  w).  A 
service  is  a  function  that  is  well-defined,  self  contained,  and  does 
not  depend  on  the  context  or  state  of  other  services.  It  easily  allows 
for  reuse  in  yet  to  be  determined  functions.  Applications  are 
designed  to  perform  a  specific  function  directly  for  the  user  or  for 
another  application.” 

US  DoD  CJCSI  6212.01  E,  15  December  2008 
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System  of  Systems 


See  “Exploring  Enterprise,  System  of  Systems,  and  System  and  Software  Architectures” 
by  Paul  C.  Clements,  SEI: 

http://www.sei.cmu.edu/library/abstracts/presentations/22ian2009webinar.cfm 

“System  of  Systems  (SoS)  Architecture 

•  A  SoS  is  a  set  or  arrangement  of  systems  that  results  when  independent  and  useful  systems  are 

integrated  into  a  larger  system  that  delivers  unique  capabilities. 

•  Varieties: 

Directed:  SoS  objectives,  management,  funding  and  authority  in  place;  systems  are 
subordinated  to  the  SoS 

Acknowledged:  SoS  objectives,  management,  funding  and  authority  in  place;  systems  retain 
their  own  management,  funding  and  authority  in  parallel  with  the  SoS 

Collaborative:  No  objectives,  management,  authority,  responsibility,  or  funding  at  the  SoS 
level;  systems  voluntarily  work  together  to  address  shared  or  common  interest 

Virtual:  Like  collaborative,  but  systems  don’t  know  about  each  other  (for  example,  the 
Internet)” 
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Contact  Information 


Presenter  /  Point  of  Contact 

Jim  Wessel  /  Bryce  Meyer 
Acquisition  Support  Program 
Telephone:  +1  908-418-0323/ 
+1  314-800-3159 
Email:  iwessel@sei.cmu.edu  / 
bmever@sei.cmu.edu 

World  Wide  Web: 

www.sei.cmu.edu 
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NO  WARRANTY 


THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING  INSTITUTE 
MATERIAL  IS  FURNISHED  ON  AN  “AS-IS"  BASIS.  CARNEGIE  MELLON  UNIVERSITY 
MAKES  NO  WARRANTIES  OF  ANY  KIND,  EITHER  EXPRESSED  OR  IMPLIED,  AS  TO 
ANY  MATTER  INCLUDING,  BUT  NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR 
PURPOSE  OR  MERCHANTABILITY,  EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM 
USE  OF  THE  MATERIAL.  CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE  ANY 
WARRANTY  OF  ANY  KIND  WITH  RESPECT  TO  FREEDOM  FROM  PATENT, 
TRADEMARK,  OR  COPYRIGHT  INFRINGEMENT. 

Use  of  any  trademarks  in  this  presentation  is  not  intended  in  any  way  to  infringe  on  the 
rights  of  the  trademark  holder. 

This  Presentation  may  be  reproduced  in  its  entirety,  without  modification,  and  freely 
distributed  in  written  or  electronic  form  without  requesting  formal  permission.  Permission 
is  required  for  any  other  use.  Requests  for  permission  should  be  directed  to  the  Software 
Engineering  Institute  at  permission@sei.cmu.edu. 

This  work  was  created  in  the  performance  of  Federal  Government  Contract  Number 
FA8721-05-C-0003  with  Carnegie  Mellon  University  for  the  operation  of  the  Software 
Engineering  Institute,  a  federally  funded  research  and  development  center.  The 
Government  of  the  United  States  has  a  royalty-free  government-purpose  license  to  use, 
duplicate,  or  disclose  the  work,  in  whole  or  in  part  and  in  any  manner,  and  to  have  or 
permit  others  to  do  so,  for  government  purposes  pursuant  to  the  copyright  license  under 
the  clause  at  252.227-7013. 
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